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DETAILED ACTION 

Response to Amendment 
A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1.17(e), was filed in this application after allowance or after an Office action under Ex 
Parte Quayle, 25 USPQ 74, 453 O.G. 213 (Comm'r Pat. 1935). Since this application is eligible 
for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been 
timely paid, prosecution in this application has been reopened pursuant to 37 CFR 1.114. 
Applicant's submission filed 9/20/2006 and 11/20/2006 have been entered. Claims 1 and 23 have 
been amended. Claims 2 and 24 have been canceled. Claims 1, 3-23, 25-44 are pending in the 
application. 



Response to Arguments 

Applicant's arguments with respect to claim 1 have been considered but are moot in view 
of the new ground(s) of rejection set forth in the present Office Action based on Watanabe et al. 
U.S. Patent No. 6,763,284 (hereinafter Watanabe) in view of Huissoon U.S. Patent No. 
6,044,308 (hereinafter Huissoon) and Roos U.S. Patent No. 6,615,112 (hereinafter Roos). 

Applicant's arguments filed 9/20/2006 have been considered and are not found 
persuasive in view of the cited prior art of record. 

Applicant argues on Page 13 that the present invention provides a method for displaying 
"abstract information that is not visible to the user" so that the user has a visual representation of 
the robot-specific conditions actually occurring. Applicant argues displaying abstract information 
that is not visible to the user. It is rather bazaar how abstract information that is not visible to the 
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user can be displayed. Applicant's argument is also in contrary to the claimed invention in which 
the robot-specific information is faded over the image of the real environment. 

Applicant also argues the present invention fades in a system of coordinates over a real 
environment image. It is not clear how a system of coordinates can be faded over a real 
environment image. The claimed language is interpreted in light of applicant's specification. 
Applicant's specification does not disclose a display of at least one robot-specific coordinate 
system. Applicant's specification in Fig. 8 discloses some rotation axes relating to the degrees of 
freedom of the robot. No coordinate axis is disclosed in relation to the world coordinate system 
or work-piece coordinate system. Because the specification is a literal translation into English, it 
is not clear what is meant by claim limitation of at least one robot-specific coordinate system 
being faded into or displayed/superimposed over said image of the real environment. In view of 
the cited prior art, Watanabe further discloses a plurality of coordinate systems including the 
world coordinate system and the coordinates of the positions of the working path-workpiece 
coordinate systems are detected and superimposed on the image of the reference work; e.g., 
column 12. lines 10-21 . 

Applicant argues that Watanabe does not teach the claim limitation of robot-specific 
information set forth in the claim 1. However, in contrary to the applicant's arguments, 
Watanabe teaches the robot-specific information . Since applicant's robot-specific information 
lakes any of a variety of forms including information related to any robot, it should be given the 
broadest reasonable interpretation. Although the claims are interpreted in light of the 
specification, limitations from the specification are not read into the claims. See In re Van 
Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). 
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Watanabe discloses the working line or the working path of the robot, which is clearly the 
robot-specific information and it has nothing to do with the work-piece; see column 7, lines 15- 
35, column 8, lines 45-65 and column 11, lines 55-67 wherein the robot working path constitute 
the teaching points of the robot to produce the movement path or a motion program of the second 
robot . 

Moreover, the position and orientations of the faces in the vicinity of the working path 
are robot specific information because the coordinates are measured in reference to the robot 
coordinate system, especially relative to the second robot. 

Moreover, Watanabe further discloses that the position or locus on which a working robot 
operates is marked on an object to be worked and a position of the marking on an image of a 
camera is detected by means for specifying points constituting a working line and the operator 
can indicate a correct point to be recognized on the camera image. These points are marked in 
order for the robot to follow the path and thus are rightfully robot specific information, as 
opposed to the work-piece related information. 

Finally, the working path is clearly faded over the image of the real environment on the 
viewing device. The image of the camera 2 or 12 is displayed on a screen of the image display 
device 3 and the detected point positions constituting the working line are displayed while being 
superimposed on this image. See Watanabe column 5, lines 10-17 and column 12, lines 45-67 
and column 13, lines 1-50. 

Applicant has not even defined the term "fading^^ in the specification and the claim. 
In the absence of any clear meaning of ^*fading^\ the term "fading^^ is given the broadest 
interpretation consistent with the specification . During patent examination, the claims are 
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given the broadest reasonable interpretation consistent with the specification. See In re Morris, 
127 F3d 1048, 44 USPQ2d 1023 (Fed. Cir. 1997). See MPEP § 2111 - § 2116.01 for case law 
pertinent to claim analysis. 

Although, it is not clear whether Watanabe discloses providing the viewing device with 
the image receiving unit, detecting an image of the real environment by the image receiving unit 
and fading computer-generated information into the image. 

However, Huissoon discloses providing the viewing device with the image (e.g., column 
6, lines 18-20), detecting an image of the real environment by the image receiving unit (column 
6, lines 1-31) and fading computer-generated information into the image (column 6, lines 30-45 
and Fig. 5a and 7b). 

It would have been obvious to have combined Huissoon and Watanabe because Huissoon 
teaches other claim limitation set forth in the claim 1 as well. Huissoon teaches the virtual edge 
shown as the broken lines are "faded" over the topographical features viewed by a structured line 
sensor wherein the virtual edges are computed generated information as claimed. 

Huissoon discloses a determination of a position and an orientation or pose of the 
reference frame S being the coordinate frame for sensor image frame (column 6, lines 5-10) and 
that robot-specific information such as the relative pose of the end-point frame E, which is the 
pose of coordinate frame E of the robot end-point with respect to a global coordinate reference 
frame of robot, is faded over the sensor image to determine the pose of the sensor frame S with 
respect to the end-point frame E using sensor measurements of a known fixture in the reference 
frame R of a calibration feature (column 6, lines 1-16). The tool center point (TCP) with respect 



Application/Control Number: 10/775,918 Page 6 

Art Unit: 2628 

to reference fixture is faded over the sensor image data (See Huissoon column 6, lines 60-67 and 
column 7, lines 1-8). The TCP is clearly robot-specific information as claimed. 

One of the ordinary skill in the art would have been motivated to do so to provide the 
viewing device for detecting an image of the real environment (Huissoon column 11, lines 4-26) 
so that the calibration markings can be displayed over the image of the real environment (Roos 
column 11, lines 40-50). 

Claims 1 and 23 are fulfilled by Saito et al. U.S. Patent No. 6,587,752. 

Saito discloses a method for fading computer-generated information into an image of the 
real environment detected by an image receiving unit located on a viewing device, the method 
comprising the steps of: 

Providing the viewing device with the image receiving unit (e.g., the vision device 44 of 
column 3 is provided with the viewing device 31); 

Detecting an image of the real environment by the image receiving unit (e.g., the vision 
device 44 captures an image of the real environment; see column 3 and column 5, Fig. 6(b)); 

Fading computer generated information into the image (Fi g. 6(b) \ wherein there is a 
determination of a position and an orientation or pose of the image receiving unit (e.g., column 
3, in accordance with instructions from the operator 5 referring to actual space images obtained 
from the vision device 44, the three-dimensional image measurement unit 21 measures 
corresponding spatial locations in the actual space. The relationship between the spatial 
coordinate system used by the three-dimensional image measurement unit 21 and the tool frame 
or the tool coordinate system given to the robot hand 42 is assumed to be known by means of 
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calibration and therefore there is a determination of a position and an orientation of the image- 
receiving unit. See Fig. 6(b), column 3 and column 6, lines 45-65 wherein the locations and 
orientations of tool coordinate system settings are modified and trajectories/paths are readily 
modified to control the robot motion) and that robot-specific information corresponding to this 
determination is faded over the image of the real environment on the viewing device (e.^., Fig. 
6(b) wherein the model space image is faded over the image of the real environment on the 
viewing device. There is a determination of the model space image by the parametric method 
using the robot-specific information such as the robot constraint conditions); at least one robot- 
specific coordinate system being faded into said image of the real environment (Fig. 6(b) shows 
the camera image overlaid with an image of the model space wherein at least one robot-specific 
coordinate system is displayed. The claimed language is interpreted in light of applicant 's 
specification. Applicant 's specification does not disclose a display of at least one robot-specific 
coordinate system. Applicant 's specification in Fig, 8 discloses some rotation axes relating to the 
degrees of freedom of the robot. No coordinate axis is disclosed in relation to the world 
coordinate system or workpiece coordinate system. Because the specification is a literal 
translation into English, it is not clear what is meant by a display of at least one robot-specific 
coordinate system). 

Claim Rejections - 35 USC § 112 
The following is a quotation of the second paragraph of 35 U.S.C. 112: 

ITic specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 
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Claims 1, 3-23, and 25-44 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 

The claims are generally narrative and indefinite, failing to conform with current U.S. 
practice. They appear to be a literal translation into English from a foreign document and are 
replete with grammatical and idiomatic errors. 

Claim Rejections - 35 USC § 112 
The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of making 
and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it 
pertains, or with which it is most nearly connected, to make and use the same and shall set forth the best mode 
contemplated by the inventor of carrying out his invention. 

Claims 1, 3-23, and 25-44 are rejected under 35 U.S.C. 112, first paragraph, as failing to 
comply with the enablement requirement. The claim(s) contains subject matter which was not 
described in the specification in such a way as to enable one skilled in the art to which it pertains, 
or with which it is most nearly connected, to make and/or use the invention. For example, Claim 
1 recites "there is a determination of a position and an orientation or pose of the image receiving 
unit and that robot-specific information corresponding to this determination is faded over the 
image of the real environment". However, applicant's specification does not support the claim 
limitation of "robot-specific information corresponding to this determination is faded over the 
image of the real envirormient." For example, in the first paragraph of Page 1 7 (applicant's 
specification), it is stated, "it is possible to establish the pose of the viewing device 1.1. 
Additionally or alternatively the pose of the viewing device 1.1 or image receiving device 1.2 
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can be obtained by a comparison of the camera image with a stored image." The pose of the 
image receiving device 1.2 is simply obtained, rather than being faded over the image of the real 
environment. The camera image is compared with a stored image. Having the determination of 
the pose of the imaging receiving unit does not imply the pose of the image receiving device 1.2 
being faded over the image of the real environment. 

Claim 1 also recites "at least one robot-specific coordinate system being faded into said 
image of the real environment." In Page 26-27 of applicant's specification, it is stated, "the 
augmented workpiece indicates to the user at which point the robot assumes the workpiece to be. 
If the robot program path does not correspond to the position of the real workpiece (e.g., due to 
positional tolerance), the user can correct the robot program in that he shifts and rotates the 
augmented workpiece together with the associated robot program path points until coincidence 
with the real workpiece is obtained.. .The augmentation of the workpiece is dependent on the 
robot path (its position changes e.g., due to the shifting of the corresponding workpiece 
coordinate system)... instead of a coordinate system the axes Al to A6 of the robot can be 
displayed, i.e., they are so faded into the image of the real robot 7 that they coincide with its 
actual axes in the image." Fading the robot program path over the real workpiece, or shifting of 
the workpiece coordinate system, or fading the axes Al to A6 into the image of the real robot 7 
is remote from "fading the at least one robot-specific coordinate system into said image of the 
real environment". There is no indication in any applicant's specification that the robot-specific 
coordinate system is faded into said image of the real environment. 

Claims 3-22 depend upon the claim 1 and are rejected due to their dependency on the 
claim 1. 
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Claim 23 also recites "a display of at least one robot-specific coordinate system." In Page 
26-27 of applicant's specification, it is stated, "the augmented workpiece indicates to the user at 
which point the robot assumes the workpiece to be. If the robot program path does not 
correspond to the position of the real workpiece (e.g., due to positional tolerance), the user can 
correct the robot program in that he shifts and rotates the augmented workpiece together with the 
associated robot program path points until coincidence with the real workpiece is obtained... The 
augmentation of the workpiece is dependent on the robot path (its position changes e.g., due to 
the shifting of the corresponding workpiece coordinate system).,, instead of a coordinate system 
the axes Al to A6 of the robot can be displayed, i.e., they are so faded into the image of the real 
robot 7 that they coincide with its actual axes in the image." Fading the robot program path over 
the real workpiece, or shifting of the workpiece coordinate system, or fading the axes Al to A6 
into the image of the real robot 7 is remote from "display of at least one robot-specific coordinate 
system". There is no indication in any applicant's specification that discloses the display of the 
robot-specific coordinate system. 

Claims 25-44 depend upon the claim 23 and are rejected due to their dependency on the 
claim 23. 

Claim Rejections - 35 USC § 112 

The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly poinling out and distinctly claiming the 
subject matter which the applicant regards as his invention. 
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Claims 1, 3-23, and 25-44 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 

For example, the claims 1, 3-23, and 25-44 recite the term ''faded over', "faded into" and 
"faded in", respectively. It is not clear whether they mean, "superimposed" or "displayed". The 
specification does not describe the term "faded" in any specific way and the specification is a 
literal translation of a foreign language into English. Clarification is required as to the term 
"faded". 

Claim Rejections - 35 USC § 102 
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

Claims 23, 25-37, 39, 41-44 are rejected under 35 U.S.C. 102(e) as being anticipated by 
Watanabe et al. U.S. Patent No. 6,763,284 (hereinafter Watanabe). 

Re Claim 23: 

Watanabe discloses device for visualizing computer-generated information in an image of 
the real environment, the device comprising: 
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A display of at least one robot-specific coordinate system {The claimed language is 

interpreted in light of applicant 's specification. Applicant 's specification does not disclose a 
display of at least one robot-specific coordinate system. Applicant's specification in Fig, 8 
discloses some rotation axes relating to the degrees of freedom of the robot. No coordinate axis 
is disclosed in relation to the world coordinate system or workpiece coordinate system. Because 
the specification is a literal translation into English, it is not clear what is meant by a display of 
at least one robot-specific coordinate system, " In view of the cited prior art, Watanabe further 
discloses a plurality of coordinate systems including the world coordinate system and the 
coordinates of the positions of the working path-workpiece coordinate systems are detected and 
superimposed on the image of the reference work; e.g., column 12. lines 10-21) : 
An image receiving device (e.g., camera of column 11-13). 

A viewing device, a determining means for determining a position and orientation or pose 
of the image receiving unit (e.g,, the 2'dimensional image pickup device 2 or 12 in Fig, 1 are 
mounted on the robot and the reference work 7 in the field of view is captured using the camera 
12 and the image is display on the image display device 3. A position to be measured by the first 
robot 5 is pointed on an image using a pointing device 4 and coordinates of the position pointed 
on the image are stored in the robot controller and thereby obtainingi determining the 
coordinates of the position. A view line corresponding to the position is obtained on the 5- 
dimensional image and the position and the orientation of the camera 12; column 11-12, 
Moreover, the 3 -dimensional positions of a sequence of the points which constitute the working 
line thus obtained are utilized as teaching points of the robot and thus constitute the robot 
specific information as teaching points of the robot to produce the movement path or a motion 
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program of the robot 5, see column 7, lines 20-35. It is also disclosed that the orientations of 
faces in the vicinity of the working line are measured in correspondence to the respective points. 
The measurement start point is selected on a working line on which an actual working is 
performed. The robot having the image pickup device 2 or 12 constitutes an image receiving 
unit) a and that robot-specific information {e.g., the working line or the working path of the robot 
is the robot-specific information; column 8, lines 50-60 and column 11, lines 55-67 wherein the 
robot workins oath is clearly the robot'Specific information, that constitute the teachin2 points 
of the robot, which is determined to produce the movement path or a motion program of the 
second robots and by a fading means for fading the determination of corresponding robot- 
specific information over the image of the real environment on the viewing device {e.g., the 
position or locus on which a workin2 robot operates is marked on an object to be worked and 
a position of the markine on an imase of a camera is detected by means for specifying points 
constituting a working line and the operator can indicate a correct point to be recognized on 
the camera imaee . Therefore, the working path is clearly faded over the image of the real 
environment on the viewing device. The image of the camera 2 or 12 is displayed on a screen of 
the image display device 3 and the detected point positions constituting the working line are 
displayed while being superimposed on this image; column 5, lines 10-17 and column 12, lines 
45-67 and column 13, lines 1-50). 



Watanabe teaches the robot-specific information . Since applicant's robot-specific 
information takes any of a variety of forms including information related to any robot, it should 
be given the broadest reasonable interpretation. Although the claims are interpreted in light of 
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the specification, limitations from the specification are not read into the claims. See In re Van 
Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

Watanabe discloses the working line or the working path of the robot, which is clearly the 
robot-specific information and it has nothing to do with the work-piece; see column 7, lines 15- 
35, column 8, lines 45-65 and column 11, lines 55-67 wherein the robot working path constitute 
the teaching points of the robot to produce the movement path or a motion program of the second 
robot . 

Moreover, the position and orientations of the faces in the vicinity of the working path 
are robot specific information because the coordinates are measured in reference to the robot 
coordinate system, especially relative to the second robot. 

Moreover, Watanabe further discloses that the position or locus on which a working robot 
operates is marked on an object to be worked and a position of the marking on an image of a 
camera is detected by means for specifying points constituting a working line and the operator 
can indicate a correct point to be recognized on the camera image. These points are marked in 
order for the robot to follow the path and thus are rightfully robot specific information, as 
opposed to the work-piece related information. 

Finally, the working path is clearly faded over the image of the real environment on the 
viewing device. The image of the camera 2 or 12 is displayed on a screen of the image display 
device 3 and the detected point positions constituting the working line are displayed while being 
superimposed on this image. See Watanabe column 5, lines 10-17 and column 12, lines 45-67 
and column 13, lines 1-50. 
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Applicant has not even defined the term ^^fading^^ in the specification and the claim. 
In the absence of any clear meaning of ^^fading^% the term **fading" is given the broadest 
interpretation consistent with the specification . During patent examination, the claims are 
given the broadest reasonable interpretation consistent with the specification. See In re Morris, 
127 F.3d 1048, 44 USPQ2d 1023 (Fed. Cir. 1997). See MPEP § 2111 - § 2116.01 for case law 
pertinent to claim analysis. 

Claim 25: 

Watanabe further discloses a plurality of coordinates being faded/superimposed on the 
reference image and the coordinates are given as coordinates to a tool coordinate system fixed on 
the distal end portion of the hand of the robot (column 2, lines 40-49). 

Claim 26: 

Watanabe further discloses the coordinate positions of the working path being 
faded/superimposed on the reference image or the working object relative to the axes of the 
three-dimensional or two-dimensional space (column 2, lines 28-49). 

Claim 27: 

Watanabe further discloses an image of a control element (working path) of a robot 
manual programmer (operator) movable in at least two dimensions (either two-dimensional 
space or three-dimensional space) in faded in /superimposed over the image of the working 
object (column 5, lines 10-17 and column 12, lines 45-67). 

Claim 28: 
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Watanabe further discloses an image of a control element such as a working path of a 
robot and the orientation of the robot hand are utilized (column 4 and column 7, lines 20-40). 
Claim 29: 

Watanabe further discloses at least one tool moved by a robot (Figs. 3-5), preferably 
several robot elements are faded into a working environment of a robot (Figs. 3-5). 

Re Claim 30: 

Watanabe further discloses teaching an attitude of a tool center point to the robot and a 
working path in relation to the reference work (column 7-8) and the thinning process of 
redundant detected points (column 7-8). 

Re Claim 31: 

Watanabe further discloses an image of a control element (working path) of a robot 
manual programmer (operator) movable in at least two dimensions (either two-dimensional 
space or three-dimensional space) in faded in /superimposed over the image of the working 
object (column 5, lines 10-17 and column 12, lines 45-67). 

Claim 32: 

Watanabe further discloses the coordinate positions of the working path being 
faded/superimposed on the reference image or the working object relative to the axes of the 
three-dimensional or two-dimensional space (column 2, lines 28-49). 

Claim 33: 

Watanabe further discloses adapting a robot working path to the position of a detected, 
real work-piece or the working object, a virtual image of the working object with a robot path 
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adapted thereto is faded in, so that by superimposing the virtual work object image with the 
image of the real object it is possible to adapt the robot path to be performed to the position of 
the real working object (column 7-8 and column 12). 
Claim 34: 

Watanabe further discloses the working area reachable by a robot and/or a permitted 
operating area is visualized on the viewing device (column 12). 
Claim 35: 

Watanabe further discloses movement corridors of a robot tool, robot hand and/or further 
robot elements are visualized on the viewing device (column 5, lines 10-30 and column 12, lines 
45-67). 

Claim 36: 

Watanabe further discloses permanent and/or instantaneous associations of at least one 
manual programmer of at least one robot are visualized (column 12, lines 45-67). 
Claim 37: 

Watanabe further discloses the position and orientation of the display are detected by 
fixed markings in space (column 12, lines 45-67). 
Claim 38: 

Watanabe further discloses the position and orientation of the viewing device are 
determined optically (column 12, lines 45-67). 
Claim 39: 

Watanabe further discloses the robot-specific information (e.g., the working line or the 
working path of the robot is the robot-specific information; column 8, lines 50-60) corresponding 



Application/Control Number: 10/775,918 Page 18 

Art Unit: 2628 

to this determination is faded over the image of the real environment on the viewing device (e.g., 
the image of the camera 2 or 12 is displayed on a screen of the image display device 3 and the 
detected point positions constituting the working line are displayed while being superimposed 
on this image; column 5, lines 10-17 and column 12, lines 45-67 and column 13, lines 1-46). 
Claims 40-44: 

Watanabe further discloses the robot-specific information {e.g., the working line or the 
working path of the robot is the robot-specific information; column 8, lines 50-60) corresponding 
to this determination is faded over the image of the real environment on the viewing device (e.g., 
the image of the camera 2 or 12 is displayed on a screen of the image display device 3 and the 
detected point positions constituting the working line are displayed while being superimposed 
on this image; column 5, lines 10-17 and column 12, lines 45-67 and column 13, lines 1-46). 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art arc 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 1, 3-15, 17, and 19-22 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Watanabe et al. U.S. Patent No. 6,763,284 (hereinafter Watanabe) in view of Huissoon U.S. 
Patent No. 6,044,308 (hereinafter Huissoon) and Roos U.S. Patent No. 6,615,112 (hereinafter 
Roos). 
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Re Claim 1: 

Watanabe discloses method for fading computer-generated information into an image of 
the real environment detected by an image receiving unit located on a viewing device, 

Wherein there is a determination of a position and an orientation or pose of the image 
receiving unit (e.g., the 2 -dimensional image pickup device 2 or 12 in Fig. 1 are mounted on the 
robot and the reference work 7 in the field of view is captured using the camera 12 and the 
image is display on the image display device 3. A position to be measured by the first robot 5 is 
pointed on an image using a pointing device 4 and coordinates of the position pointed on the 
image are stored in the robot controller and thereby obtainingi determining the coordinates of 
the position, A view line corresponding to the position is obtained on the 3-dimensional image 
and the position and the orientation of the camera 12; column 11-12. Moreover, the 3- 
dimensional positions of a sequence of the points which constitute the working line thus obtained 
are utilized as teaching points of the robot and thus constitute the robot specific information as 
teaching points of the robot to produce the movement path or a motion program of the robot 5, 
see column 7, lines 20-35. It is also disclosed that the orientations of faces in the vicinity of the 
working line are measured in correspondence to the respective points. The measurement start 
point is selected on a working line on which an actual working is performed. The robot having 
the image pickup device 2 or 12 constitutes an image receiving unit) and that robot-specific 
information (e.g., the working line or the working path of the robot is the robot-specific 
information; column 8, lines 50-60 and column 11, lines 55-67 wherein the robot working path 
is clearly the robot'Specific information, that constitute the teaching points of the robot to 
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produce the movement path or a motion prosram of the second robot) corresponding to this 
determination is faded over the image of the real environment on the viewing device {e.g., the 
position or locus on which a workin2 robot operates is marked on an object to be worked and 
a position of the markins on an ima2e of a camera is detected by means for specifvins points 
constituting a workin2 line and the operator can indicate a correct point to be recognized on 
the camera imase . Therefore, the working path is clearly faded over the image of the real 
environment on the viewing device. The image of the camera 2 or 12 is displayed on a screen of 
the image display device 3 and the detected point positions constituting the working line are 
displayed while being superimposed on this image; column 5, lines 10-17 and column 12, lines 
45-67 and column 13, lines 1-50); at least one robot-specific coordinate system being faded into 
said image of the real environment (The claimed language is interpreted in light of applicant 's 
specification. Applicant 's specification does not disclose a display of at least one robot-specific 
coordinate system. Applicant 's specification in Fig. 8 discloses some rotation axes relating to the 
degrees of freedom of the robot. No coordinate axis is disclosed in relation to the world 
coordinate system or workpiece coordinate system. Because the specification is a literal 
translation into English, it is not clear what is meant by claim limitation of at least one robot- 
specific coordinate system being faded into or display edi superimposed over said image of the 
real environment. In view of the cited prior art, Watanabe further discloses a plurality of 
coordinate systems including the world coordinate system and the coordinates of the positions of 
the working path-workpiece coordinate systems are detected and superimposed on the image of 
the reference work; e.g., column 12, lines 10-21) . 
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Watanabe teaches the robot-specific information . Since applicant's robot-specific 
information takes any of a variety of forms including information related to any robot, it should 
be given the broadest reasonable interpretation. Although the claims are interpreted in light of 
the specification, limitations from the specification are not read into the claims. See In re Van 
Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

Watanabe discloses the working line or the working path of the robot, which is clearly the 
robot-specific information and it has nothing to do with the work-piece; see column 7, lines 15- 
35, column 8, lines 45-65 and column 11, lines 55-67 wherein the robot working path constitute 
the teaching points of the robot to produce the movement path or a motion program of the second 
robot. 

Moreover, the position and orientations of the faces in the vicinity of the working path 
are robot specific information because the coordinates are measured in reference to the robot 
coordinate system, especially relative to the second robot. 

Moreover, Watanabe further discloses that the position or locus on which a working robot 
operates is marked on an object to be worked and a position of the marking on an image of a 
camera is detected by means for specifying points constituting a working line and the operator 
can indicate a correct point to be recognized on the camera image. These points are marked in 
order for the robot to follow the path and thus are rightfully robot specific information, as 
opposed to the work-piece related information. 

Finally, the working path is clearly faded over the image of the real environment on the 
viewing device. The image of the camera 2 or 12 is displayed on a screen of the image display 
device 3 and the detected point positions constituting the working line are displayed while being 
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superimposed on this image. See Watanabe column 5, lines 10-17 and column 12, lines 45-67 
and column 13, lines 1-50. 

Applicant has not even defined the term ^^fading** in the specification and the claim. 
In the absence of any clear meaning of '^fading'\ the term ^^fading^^ is given the broadest 
interpretation consistent with the specification . During patent examination, the claims are 
given the broadest reasonable interpretation consistent with the specification. See In re Morris, 
127 F.3d 1048, 44 USPQ2d 1023 (Fed. Cir. 1997). See MPEP § 2111 - § 2116.01 for case law 
pertinent to claim analysis. 

It is not clear whether Watanabe discloses providing the viewing device with the image 
receiving unit, detecting an image of the real environment by the image receiving unit and fading 
computer-generated information into the image. 

However, Huissoon discloses providing the viewing device with the image (e.g., column 
6, lines 18-20), detecting an image of the real environment by the image receiving unit (column 
6, lines 1-31) and fading computer-generated information into the image (column 6, lines 30-45 
and Fig. 5a and 7b). 

It would have been obvious to have combined Huissoon and Watanabe because Huissoon 
teaches other claim limitation set forth in the claim 1 as well. Huissoon teaches the virtual edge 
shown as the broken lines are "faded" over the topographical features viewed by a structured line 
sensor wherein the virtual edges are computed generated information as claimed. 

Huissoon discloses a determination of a position and an orientation or pose of the 
reference frame S being the coordinate frame for sensor image frame (column 6, lines 5-10) and 
that robot-specific information such as the relative pose of the end-point frame E, which is the 
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pose of coordinate frame E of the robot end-point with respect to a global coordinate reference 
frame of robot, is faded over the sensor image to determine the pose of the sensor frame S with 
respect to the end-point frame E using sensor measurements of a known fixture in the reference 
frame R of a calibration feature (column 6, lines 1-16). The tool center point (TCP) with respect 
to reference fixture is faded over the sensor image data (See Huissoon column 6, lines 60-67 and 
column 7, lines 1-8). The TCP is clearly robot-specific information as claimed. 

One of the ordinary skill in the art would have been motivated to do so to provide the 
viewing device for detecting an image of the real environment (Huissoon column 11, lines 4-26) 
so that the calibration markings can be displayed over the image of the real environment (Roos 
column 11, lines 40-50). 

Claim 3: 

Watanabe further discloses a plurality of coordinates being faded/superimposed on the 
reference image and the coordinates are given as coordinates to a tool coordinate system fixed on 
the distal end portion of the hand of the robot (column 2, lines 40-49). 

Claim 4: 

Watanabe further discloses the coordinate positions of the working path being 
faded/superimposed on the reference image or the working object relative to the axes of the 
three-dimensional or two-dimensional space (column 2, lines 28-49). 

Claim 5: 

Watanabe further discloses an image of a control element (working path) of a robot 
manual programmer (operator) movable in at least two dimensions (either two-dimensional 
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space or three-dimensional space) in faded in /superimposed over the image of the working 

object (column 5, lines 10-17 and column 12, lines 45-67). 
Claim 6: 

Watanabe further discloses an image of a control element such as a working path of a 
robot and the orientation of the robot hand are utilized (column 4 and column 7, lines 20-40). 
Claim 7: 

Watanabe further discloses at least one tool moved by a robot (Figs. 3-5), preferably 
several robot elements are faded into a working environment of a robot (Figs. 3-5). 
Re Claim 8: 

Watanabe further discloses teaching an attitude of a tool center point to the robot and a 
working path in relation to the reference work (column 7-8) and the thinning process of 
redundant detected points (column 7-8). 

Re Claim 9: 

Watanabe further discloses an image of a control element (working path) of a robot 
manual programmer (operator) movable in at least two dimensions (either two-dimensional 
space or three-dimensional space) in faded in /superimposed over the image of the working 

object {column 5, lines 10-17 and column 12, lines 45-67). 

Claim 10: 

Watanabe further discloses the coordinate positions of the working path being 
faded/superimposed on the reference image or the working object relative to the axes of the 
three-dimensional or two-dimensional space (column 2, lines 28-49). 
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Claim 11: 

Watanabe further discloses adapting a robot working path to the position of a detected, 
real work-piece or the working object, a virtual image of the working object with a robot path 
adapted thereto is faded in, so that by superimposing the virtual work object image with the 
image of the real object it is possible to adapt the robot path to be performed to the position of 
the real working object (column 7-8 and column 12). 

Claim 12: 

Watanabe further discloses the working area reachable by a robot and/or a permitted 
operating area is visualized on the viewing device (column 12). 
Claim 13: 

Watanabe further discloses movement corridors of a robot tool, robot hand and/or further 
robot elements are visualized on the viewing device (column 5, lines 10-30 and column 12, lines 
45-67). 

Claim 14: 

Watanabe further discloses permanent and/or instantaneous associations of at least one 
manual programmer of at least one robot are visualized (column 12, lines 45-67). 
Claim 15: 

Watanabe further discloses the position and orientation of the display are detected by 
fixed markings in space (column 12, lines 45-67). 
Claim 17: 

Watanabe further discloses the position and orientation of the viewing device are 
determined optically (column 12, lines 45-67). 
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Claim 19: 

Watanabe further discloses the robot-specific information (e,g., the working line or the 
working path of the robot is the robot-specific information; column 8, lines 50-60) corresponding 
to this determination is faded over the image of the real environment on the viewing device (e.g., 
the image of the camera 2 or 12 is displayed on a screen of the image display device 3 and the 
detected point positions constituting the working line are displayed while being superimposed 
on this image; column 5, lines 10-17 and column 12, lines 45-67 and column 13, lines 1-46). 

Claims 20-22: 

Watanabe further discloses the robot-specific information (e.g., the working line or the 
working path of the robot is the robot-specific information; column 8, lines 50-60) corresponding 
to this determination is faded over the image of the real environment on the viewing device (e.g., 
the image of the camera 2 or 12 is displayed on a screen of the image display device 3 and the 
detected point positions constituting the working line are displayed while being superimposed 
on this image; column 5, lines 10-17 and column 12, lines 45-67 and column 13, lines 1-46). 

Claim 16 is rejected under 35 U.S.C. 103(a) as being unpatentable over Watanabe et al. 
U.S. Patent No. 6,763,284 (hereinafter Watanabe) in view of Huissoon U.S. Patent No. 
6,044,308 (hereinafter Huissoon) and Roos U.S. Patent No. 6,615,112 (hereinafter Roos). 



Claim 16: 
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The claims further recite markings are detected by radio receivers. Watanabe, Huissoon 
and Roos are silent to the claim limitation. 

It would have been obvious to have incorporated radio receivers to collect the working 
path information or to serve as an image pickup device because Watanabe discloses a general 
image pickup device such as a camera or an optical receiver for collecting the path information 
(Watanabe Figs. 1-5) and thereby suggesting the claim limitation of an image pickup device such 
as the radio receivers. Moreover, the radio receiver can be used in replace with the optical 
receiver as an alternative image and information collection device. 

One of the ordinary skill in the art would have been motivated to use an alternative 
receiver to collect the robot working path information and to pick up the image information 
when necessary (column 9-10). 

Claim 18 is rejected under 35 U.S.C. 103(a) as being unpatentable over Watanabe et al. 
U.S. Patent No. 6,763,284 (hereinafter Watanabe) in view of Mizuno et al. U.S. Patent No. 
5,876,325 (hereinafter Mizuno), Huissoon U.S. Patent No. 6,044,308 (hereinafter Huissoon) and 
Roos U.S. Patent No. 6,615,112 (hereinafter Roos). 

Claim 18: 

The claims further recite data spectacles to be worn by a user for displaying the robot 
information. Watanabe, Huissoon and Roos are silent to the claim limitation. 

However, Mizuno discloses HMD for displaying the robot information (Mizuno Fig. 28 
and 35). 
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It would have been obvious to have incorporated HMD to display the robot information 
because Watanabe discloses a display device connected to the robot for collecting the robot 
specific information (Watanabe Figs. 1-5) and Mizuno discloses HMD coupled to the robot 
manipulators for collecting the robot specific information (Mizuno Figs. 28 and 35) and therefore 
an alternative display device can be used to collect the robot specific information. 

One of the ordinary skill in the art would have been motivated to use an alternative 
display device such as an HMD so that an operator armed with HMD can directly see the work 
performed while specifying the points along the working path (Watanabe column 12, lines 45- 
67). 

Claim 38 is rejected under 35 U.S.C. 103(a) as being unpatentable over Watanabe et al. 
U.S. Patent No. 6,763,284 (hereinafter Watanabe). 

Claim 38: 

The claims further recite markings are detected by radio receivers. Watanabe is silent to 
the claim limitation. 

It would have been obvious to have incorporated radio receivers to collect the working 
path information or to serve as an image pickup device because Watanabe discloses a general 
image pickup device such as a camera or an optical receiver for collecting the path information 
(Watanabe Figs. 1-5) and thereby suggesting the claim limitation of an image pickup device such 
as the radio receivers. Moreover, the radio receiver can be used in replace with the optical 
receiver as an alternative image and information collection device. 
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One of the ordinary skill in the art would have been motivated to use an alternative 
receiver to collect the robot working path information and to pick up the image information 
when necessary (column 9-10). 

Claim 40 is rejected under 35 U.S.C. 103(a) as being unpatentable over Watanabe et al. 
U.S. Patent No. 6,763,284 (hereinafter Watanabe) in view of Mizuno et al. U.S. Patent No. 
5,876,325 (hereinafter Mizuno). 

Claim 40: 

The claims further recite data spectacles to be worn by a user for displaying the robot 
information. Watanabe is silent to the claim limitation. 

However, Mizuno discloses HMD for displaying the robot information (Mizuno Fig. 28 
and 35). 

It would have been obvious to have incorporated HMD to display the robot information 
because Watanabe discloses a display device connected to the robot for collecting the robot 
specific information (Watanabe Figs. 1-5) and Mizuno discloses HMD coupled to the robot 
manipulators for collecting the robot specific information (Mizuno Figs. 28 and 35) and therefore 
an alternative display device can be used to collect the robot specific information. 

One of the ordinary skill in the art would have been motivated to use an alternative 
display device such as an HMD so that an operator armed with HMD can directly see the work 
performed while specifying the points along the working path (Watanabe column 12, lines 45- 
67). 
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Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 

patent by another filed in the United Slates before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

Claims 1 and 23 are rejected under 35 U.S.C. 102(e) as being anticipated by Saito et al. 
U.S. Patent No. 6,587,752 (hereinafter Saito). 
Re Claim 1: 

Saito discloses a method for fading computer-generated information into an image of the 
real environment detected by an image receiving unit located on a viewing device, the method 
comprising the steps of: 

Providing the viewing device with the image receiving unit (eg., the vision device 44 of 
column 3 is provided with the viewing device 31); 

Detecting an image of the real environment by the image receiving unit (e.g., the vision 
device 44 captures an image of the real environment; see column 3 and column 5, Fig. 6(b)); 

Fading computer generated information into the image ( Fig. 6(b) \ wherein there is a 
determination of a position and an orientation or pose of the image receiving unit {e.g., column 
3, in accordance with instructions from the operator 5 referring to actual space images obtained 
from the vision device 44, the three-dimensional image measurement unit 21 measures 
corresponding spatial locations in the actual space. The relationship between the spatial 
coordinate system used by the three-dimensional image measurement unit 21 and the tool frame 
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or the tool coordinate system given to the robot hand 42 is assumed to be known by means of 
calibration and therefore there is a determination of a position and an orientation of the image- 
receiving unit. See Fig. 6(b), column 3 and column 6, lines 45-65 wherein the locations and 
orientations of tool coordinate system settings are modified and trajectories/paths are readily 
modified to control the robot motion) and that robot-specific information corresponding to this 
determination is faded over the image of the real environment on the viewing device (e.g., Fig. 
6(b) wherein the model space image is faded over the image of the real environment on the 
viewing device. There is a determination of the model space image by the parametric method 
using the robot-specific information such as the robot constraint conditions); at least one robot- 
specific coordinate system being faded into said image of the real environment (Fig. 6(b) shows 
the camera image overlaid with an image of the model space wherein at least one robot-specific 
coordinate system is displayed. The claimed language is interpreted in light of applicant 's 
specification. Applicant's specification does not disclose a display of at least one robot-specific 
coordinate system. Applicant's specification in Fig. 8 discloses some rotation axes relating to the 
degrees of freedom of the robot. No coordinate axis is disclosed in relation to the world 
coordinate system or workpiece coordinate system. Because the specification is a literal 
translation into English, it is not clear what is meant by a display of at least one robot-specific 
coordinate system). 

Re Claim 23: 

Saito discloses device for visualizing computer-generated information in an image of the 
real environment, the device comprising: 
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A display of at least one robot-specific coordinate system (Fig. 6(b) shows the camera 
image overlaid with an image of the model space wherein at least one robot-specific coordinate 
system is displayed. The claimed language is interpreted in light of applicant 's specification. 
Applicant's specification does not disclose a display of at least one robot-specific coordinate 
system. Applicant 's specification in Fig, 8 discloses some rotation axes relating to the degrees of 
fi'eedom of the robot. No coordinate axis is disclosed in relation to the world coordinate system 
or workpiece coordinate system. Because the specification is a literal translation into English, it 
is not clear what is meant by a display of at least one robot-specific coordinate system); 

An image receiving device (e.g., camera or the vision device 44 of column 3). 

A viewing device, a determining means for determining a position and orientation or pose 
of the image receiving unit (e.g., column 3, in accordance with instructions from the operator 5 
referring to actual space images obtained from the vision device 44, the three-dimensional image 
measurement unit 21 measures corresponding spatial locations in the actual space. The 
relationship between the spatial coordinate system used by the three-dimensional image 
measurement unit 21 and the tool frame or the tool coordinate system given to the robot hand 42 
is assumed to be known by means of calibration) a and that robot-specific information (e.g.. Fig. 
6(b), column 3 and column 6, lines 45-65 wherein the locations and orientations of tool 
coordinate system settings are modified and trajectories/paths are readily modified to control the 
robot motion) and by a fading means for fading the determination of corresponding robot- 
specific information over the image of the real environment on the viewing device (e.g.. Fig. 
6(b) wherein the model space image is faded over the image of the real environment on the 
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viewing device. There is a determination of the model space image by the parametric method 
using the robot-specific information such as the robot constraint conditions). 



Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (571) 272-7665. 
The examiner can normally be reached on 8:00 - 6:30 (Mon-Thu). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung can be reached on (571) 272-7794. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public.PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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